Add new TransactionDetails level #27601

CriesofCarrots · 2022-09-06T05:22:00Z

Problem

Responses for getBlock rpc requests can be huge. The TransactionDetails parameter enables reducing returned transactions to just signatures, or eliminating the details altogether. But any consumer that needs to track account balance changes in each block (an exchange, eg) has to pull the full details, even though they don't need instruction data, inner-instruction or log details, etc.

Summary of Changes

Add a new TransactionDetails level that slims each transaction in a block to just pre/postBalances (sol and token), signatures, and the list of accounts.

CriesofCarrots · 2022-09-06T18:27:11Z

Example:

$ curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0","id":1,"method":"getBlock","params":[160257046,{"transactionDetails":"accounts","rewards":false}]}' <devnet_node> | jq

{
  "jsonrpc": "2.0",
  "result": {
    "blockHeight": 150808359,
    "blockTime": 1662485419,
    "blockhash": "CdzN3T7ERcxjqk8PxEqq7mErxDL6CoD8CDEyp78NbJz5",
    "parentSlot": 160257045,
    "previousBlockhash": "3E5Z6vt3Jvx6hHPaP15gXhJTpQo4ABTaKgHX9zAf8KkT",
    "transactions": [
      {
        "meta": {
          "err": null,
          "fee": 5000,
          "postBalances": [
            204375923219118,
            100000000000,
            1169280,
            143487360,
            1
          ],
          "postTokenBalances": [],
          "preBalances": [
            204375923224118,
            100000000000,
            1169280,
            143487360,
            1
          ],
          "preTokenBalances": [],
          "status": {
            "Ok": null
          }
        },
        "transaction": {
          "accountKeys": [
            {
              "pubkey": "dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS",
              "signer": true,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "vgcDar2pryHvMgPkKaZfh8pQy4BJxv7SpwUG7zinWjG",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "SysvarC1ock11111111111111111111111111111111",
              "signer": false,
              "source": "transaction",
              "writable": false
            },
            {
              "pubkey": "SysvarS1otHashes111111111111111111111111111",
              "signer": false,
              "source": "transaction",
              "writable": false
            },
            {
              "pubkey": "Vote111111111111111111111111111111111111111",
              "signer": false,
              "source": "transaction",
              "writable": false
            }
          ],
          "signatures": [
            "2jnhgWUBWsFk73gPwkcXBGNS2sKquivC5rjFWrxfHyXbbsxcJxLRJP3FSUwZG2KtinVytJ17sBq7Et2R6PkJHEfp"
          ]
        }
      },
      {
        "meta": {
          "err": null,
          "fee": 5000,
          "postBalances": [
            974108077111,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            1169280,
            1141440
          ],
          "postTokenBalances": [],
          "preBalances": [
            974108082111,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            23942400,
            1169280,
            1141440
          ],
          "preTokenBalances": [],
          "status": {
            "Ok": null
          }
        },
        "transaction": {
          "accountKeys": [
            {
              "pubkey": "5U3bH5b6XtG99aVWLqwVzYPVpQiFHytBD68Rz2eFPZd7",
              "signer": true,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "7csi4gj91mycrtkhjdiKjJwoxavP84PMxnrtkqGafkVZ",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "38xoQ4oeJCBrcVvca2cGk7iV1dAfrmTR1kmhSCJQ8Jto",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "JAe1wPF7HgueCbqVcM5qfjskmURgyDhptKhTnC1PahaW",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "AUKjh1oVPZyudi3nzYSsdZxSjq42afUCvsdbKFc5CbD",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "Gvm85Pbjq4Tv7qyaS4y9ZMqCdY3nynGDBFYAu7mjPoGM",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "4NbqLAy4LjhZ7imWbyTuHo4EMSAVWTVDFS6Kpwz15Kyc",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "4GqTjGm686yihQ1m1YdTsSvfm4mNfadv6xskzgCYWNC5",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "4L6YhY8VvUgmqG5MvJkUJATtzB2rFqdrJwQCmFLv4Jzy",
              "signer": false,
              "source": "transaction",
              "writable": true
            },
            {
              "pubkey": "SysvarC1ock11111111111111111111111111111111",
              "signer": false,
              "source": "transaction",
              "writable": false
            },
            {
              "pubkey": "gSbePebfvPy7tRqimPoVecS2UsBvYv46ynrzWocc92s",
              "signer": false,
              "source": "transaction",
              "writable": false
            }
          ],
          "signatures": [
            "2DrtzGnenMJzn6BCqMvaJHVoW6j5BSzTf7nM1tAFcRkXApNbR1uNdZyPJwYUYv92iqfiVrLnNFYaJG4z71R6C3Mi"
          ]
        }
      },
      ...
  "id": 1
}

CriesofCarrots · 2022-09-06T18:28:19Z

Wydt about not returning the empty metadata fields? Too breaky? I split that into its own commit, so easy to revert if need be.

jstarry · 2022-09-07T14:24:53Z

Wydt about not returning the empty metadata fields? Too breaky? I split that into its own commit, so easy to revert if need be.

I'm in favor of it for this new details level but it would be a breaking change if the other responses are impacted because some sdk's and clients may expect those fields to be always present (though the web3 and rust sdk's wouldn't be impacted from what I can tell)

jstarry

Direction looks good to me!

transaction-status/src/lib.rs

CriesofCarrots · 2022-09-07T21:49:59Z

I'm in favor of it for this new details level but it would be a breaking change if the other responses are impacted because some sdk's and clients may expect those fields to be always present

Okay, please see the last commit for an implementation that preserve current serialization for existing parameter inputs, but only includes the necessary fields for TransactionDetails::Accounts.
I was hoping there would be some slick little way to make the skip_serializing_if conditional, but the method can only inspect the field itself, so that approach required wrapping each field in an enum and got fairly ugly.

Wdyt @jstarry?

jstarry · 2022-09-07T22:10:39Z

The wrapper looks ok I think but I'm worried that approach leads to too many struct permutations if in the future we want to selectively enable / disable certain fields. It would be nice if we had more granular control over what gets serialized. What do you think about replacing some usages of Option with a new enum which has variants for skipping deserialization, returning null, and returning the value?

CriesofCarrots · 2022-09-07T22:15:33Z

What do you think about replacing some usages of Option with a new enum which has variants for skipping deserialization, returning null, and returning the value?

Oh hmm, adding a wrapper enum and removing the Option is an interesting idea. I'll try that out and if it seems feasible, push as a separate PR to rebase this one on, so we can see what it looks like in isolation.

willhickey · 2022-09-13T12:57:09Z

Updated labels to v1.13 & v1.14 which are effectively the new v1.10 and v1.11 See discussion on Discord in #releng and #proj-quic-tpu for more context.
https://discord.com/channels/428295358100013066/910937142182682656/1019051275318476881

CriesofCarrots · 2022-09-14T05:47:59Z

Updated, needs rebase on #27676

* Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb) # Conflicts: # docs/src/developing/clients/jsonrpc-api.md

* Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb)

* Add new TransactionDetails level (#27601) * Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb) * Fix doc conflict Co-authored-by: Tyera Eulberg <tyera@solana.com>

Add new TransactionDetails level (#27601) * Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb) Co-authored-by: Tyera Eulberg <tyera@solana.com>

* Add new TransactionDetails level (#27601) * Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb) * Fix doc conflict

steveluscher · 2022-11-30T21:07:21Z

Toss me a note (eg. #28999) whenever you see this happening so that I can update the JavaScript client API.

In the near future we're going to make this sort of thing impossible to miss by codegenerating the JS types from the Rust implementation, but until then I could use the pings! <3

CriesofCarrots force-pushed the tx-details branch from 95a817d to da0dcd7 Compare September 6, 2022 17:06

CriesofCarrots marked this pull request as ready for review September 6, 2022 17:06

CriesofCarrots requested a review from jstarry September 6, 2022 17:06

CriesofCarrots added v1.10 labels Sep 6, 2022

CriesofCarrots force-pushed the tx-details branch from 6e13229 to 41be8e7 Compare September 6, 2022 18:18

jstarry reviewed Sep 7, 2022

View reviewed changes

transaction-status/src/lib.rs Show resolved Hide resolved

transaction-status/src/lib.rs Show resolved Hide resolved

CriesofCarrots force-pushed the tx-details branch 2 times, most recently from 7179a78 to 3bddfa1 Compare September 7, 2022 21:28

CriesofCarrots mentioned this pull request Sep 8, 2022

Add serialization-control enum for RPC Options #27676

Merged

willhickey added v1.13 and removed v1.10 labels Sep 13, 2022

CriesofCarrots force-pushed the tx-details branch from 3bddfa1 to 750a083 Compare September 14, 2022 05:47

CriesofCarrots force-pushed the tx-details branch from 750a083 to 8f5472b Compare September 14, 2022 19:19

Tyera Eulberg added 4 commits September 14, 2022 17:07

Add new transaction details level

77e53a1

Dedupe common code

501426d

Update docs

f8ef730

Respect showRewards parameter in tx meta

3b5bd89

CriesofCarrots force-pushed the tx-details branch from 8f5472b to 3b5bd89 Compare September 14, 2022 23:07

CriesofCarrots requested a review from jstarry September 14, 2022 23:41

jstarry approved these changes Sep 15, 2022

View reviewed changes

CriesofCarrots merged commit 973a5fb into solana-labs:master Sep 15, 2022

mergify bot mentioned this pull request Sep 15, 2022

Add new TransactionDetails level (backport #27601) #27804

Merged

mergify bot mentioned this pull request Sep 15, 2022

Add new TransactionDetails level (backport #27601) #27805

Merged

mergify bot pushed a commit that referenced this pull request Sep 15, 2022

Add new TransactionDetails level (#27601)

55f9fa0

* Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb)

CriesofCarrots pushed a commit that referenced this pull request Sep 15, 2022

Add new TransactionDetails level (#27601)

819726f

* Add new transaction details level * Dedupe common code * Update docs * Respect showRewards parameter in tx meta (cherry picked from commit 973a5fb)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new TransactionDetails level #27601

Add new TransactionDetails level #27601

CriesofCarrots commented Sep 6, 2022

CriesofCarrots commented Sep 6, 2022

CriesofCarrots commented Sep 6, 2022

jstarry commented Sep 7, 2022

jstarry left a comment

CriesofCarrots commented Sep 7, 2022

jstarry commented Sep 7, 2022

CriesofCarrots commented Sep 7, 2022

willhickey commented Sep 13, 2022

CriesofCarrots commented Sep 14, 2022

steveluscher commented Nov 30, 2022

Add new TransactionDetails level #27601

Add new TransactionDetails level #27601

Conversation

CriesofCarrots commented Sep 6, 2022

Problem

Summary of Changes

CriesofCarrots commented Sep 6, 2022

CriesofCarrots commented Sep 6, 2022

jstarry commented Sep 7, 2022

jstarry left a comment

Choose a reason for hiding this comment

CriesofCarrots commented Sep 7, 2022

jstarry commented Sep 7, 2022

CriesofCarrots commented Sep 7, 2022

willhickey commented Sep 13, 2022

CriesofCarrots commented Sep 14, 2022

steveluscher commented Nov 30, 2022